Interactive Vision from the Top Down: Interactional Structure Applied to the Identification and Interpretation of Visual Interactive Behaviour

This paper considers the construction of machine vision systems for tracking interaction, informed by work from the human sciences1. The emphasis is on the construction of what we refer to as ‘global level systems’ or ‘global systems’ for short, which treat the unit of interaction as being relatively large (e.g. the script for a greeting, proceeding from an initial distant salutation to the final close salutation). These larger structures seem more appropriate for interactions that are formalized to some degree i.e. those that tend to follow a particular sequence of steps with reasonable consistency. We note that there is a continuum of formality in interaction (Hutchby and Wooffitt, 1998), from very informal interactions (e.g. play) through semi-formal interactions (e.g. greetings) to very formal interactions tending to follow a script very closely (e.g. interrogation of a witness by a lawyer). Informal interactions are harder to represent with a script-like structure: for such interactions it seems that a lower-level approach is more appropriate and we briefly consider this ‘local level’ approach here. The focus of this paper, however, is on the higher level global systems and the bulk of the paper is concerned with such systems. In the following section we discuss some of the basic concepts involved in this work, providing definitions of terms that will be used in the paper and background information to provide a context for the rest of the paper. We then discuss interactive behaviour from the perspective of first global, then local structures. We then outline the basic elements of a system designed to track interactions using the global perspective, considering various kinds of systems as we do so. Finally, we discuss evaluation methods for systems of this nature and outline an experimental set-up that could be used to test the ideas presented in this paper.